Having read the authors' response and the other reviews, I think I may have gotten a little overly excited about this paper, but I still think it is innovative and significant. In fact, I found the relationship between the two clusters in Figures 6a and 6b more convincing than I did originally. However, I am dropping my score a notch because, while I really like this paper, I believe it is in the top 50%, not the top 15% of NIPS papers. This is again, because the analysis is not as rigorous as one would like. Also, after having read the author's response and the other reviews, I have edited my review a bit below.

This paper takes a creative idea of using the feature spaces from many different computer vision task networks to help understand fMRI data from different brain regions. Reviewers find the idea interesting but the results are not particularly clear/definitive or surprising.

Convolutional neural networks (CNNs) trained for object classification have been widely used to account for visually-driven neural responses in both human and primate brains. However, because of the generality and complexity of object classification, despite the effectiveness of CNNs in predicting brain activity, it is difficult to draw specific inferences about neural information processing using CNN-derived representations. To address this problem, we used learned representations drawn from 21 computer vision tasks to construct encoding models for predicting brain responses from BOLD5000---a large-scale dataset comprised of fMRI scans collected while observers viewed over 5000 naturalistic scene and object images. Encoding models based on task features predict activity in different regions across the whole brain. Features from 3D tasks such as keypoint/edge detection explain greater variance compared to 2D tasks---a pattern observed across the whole brain.

Convolutional neural networks (CNNs) trained for object classification have been widely used to account for visually-driven neural responses in both human and primate brains. However, because of the generality and complexity of object classification, despite the effectiveness of CNNs in predicting brain activity, it is difficult to draw specific inferences about neural information processing using CNN-derived representations. To address this problem, we used learned representations drawn from 21 computer vision tasks to construct encoding models for predicting brain responses from BOLD5000---a large-scale dataset comprised of fMRI scans collected while observers viewed over 5000 naturalistic scene and object images. Encoding models based on task features predict activity in different regions across the whole brain. Features from 3D tasks such as keypoint/edge detection explain greater variance compared to 2D tasks---a pattern observed across the whole brain.